partitioned write support #353
Conversation
pyiceberg/table/__init__.py
Outdated
| arrow_table_partition: pa.Table | ||
|
|
||
|
|
||
| def get_partition_sort_order(partition_columns: list[str], reverse: bool = False) -> dict[str, Any]: |
There was a problem hiding this comment.
| def get_partition_sort_order(partition_columns: list[str], reverse: bool = False) -> dict[str, Any]: | |
| def _get_partition_sort_order(partition_columns: list[str], reverse: bool = False) -> dict[str, Any]: |
This is a nit: should we use _single_leading_underscore to indicate that these are internal functions according to PEP-8?
There was a problem hiding this comment.
Yes, thanks for the reminder! More refactoring is on the way.
jqin61
force-pushed
the
partitioned-write
branch
from
d03f058
to
b74521e
Compare
pyiceberg/manifest.py
Outdated
| @@ -308,6 +308,7 @@ def data_file_with_partition(partition_type: StructType, format_version: Literal | |||
| field_id=field.field_id, | |||
| name=field.name, | |||
| field_type=partition_field_to_data_file_partition_field(field.field_type), | |||
| required=False | |||
There was a problem hiding this comment.
Default value is True, which breaks the avro writer encoding manifest entry when the partition value is null. (Because the attribute of required being True in the nested field of the Avro schema will be used by the writer to infer and choose a non-optional field writer that could not encode None value.)
This is for encoding only, so set it to False for now.
TODO: Set it properly according to table schema.
jqin61
force-pushed
the
partitioned-write
branch
2 times, most recently
from
805c19c
to
7a33b23
Compare
jqin61
force-pushed
the
partitioned-write
branch
from
63a68b4
to
0e57e95
Compare
jqin61
force-pushed
the
partitioned-write
branch
from
abc4ca3
to
0a199a3
Compare
|
As discussed in the monthly community sync, this will be broken down into 4 prs of:
|
Todo